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Abstract 

We consider a solution of the atmospheric neutrino problem based on oscil- 
lations of muon neutrinos to sterile neutrinos: <-> i^g- The zenith angle 
(G) dependences of the neutrino and upward-going muon fluxes in presence 
of these oscillations are studied. The dependences have characteristic form 
with two dips: at cosG = — 0.6 — 0.2 and cos© = — 1.0-; — 0.8. The latter 
dip is due to parametric resonance in oscillations of neutrinos which cross the 
core of the earth. A comparison of predictions with data from the MACRO, 
Baksan and Super-Kamiokande experiments is given. 
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1. Introduction 

Recently, the Super-Kamiokande Jll and Soudan experiments liave further confirmed 
an existence of the atmospheric neutrino problem ^j^. Moreover, observations of the zenith 
angle as well as, L/E (distance /energy) dependences of the muon neutrino deficit strongly 
indicate an oscillation solution of the problem. 

Oscillations should give an observable effect in the zenith angle (G-)dependence of the flux 
of the upward-going muons produced by high energy neutrinos. However, the experimental 
situation is not yet clear. A peculiar 6-dependence has been observed by MACRO experi- 
ment 10. The ratio of the observed and expected fluxes as the function of 6 has two dips. 
The wide dip is in the range cos = —0.6 ^ —0.2 with minimum at cos O = —0.4 ^ —0.5, 
and the narrow dip is at cosO = —1.0 ^ — 0.8 (vertical bins). In the narrow dip the ob- 
served flux is almost two times smaller than the expected one. Between these two dips 
(cosO ~ —0.8) the flux is close to the value without oscillations. It is important that for 
the vertical bins there is the best understanding of the acceptance and efficiencies. The 
probability of the no oscillation hypothesis is smaller than 1%, and inclusion of — > Ur 
oscillations does not raise the probability in a significant way 0]. Baksan experiment has 

similar threshold for muon detection. The data also show a wide dip at cos 6 = —0.6 -. 0.2 

similar to that in MACRO data. However, no significant deficit has been observed in the 
vertical bins. The Super-Kamiokande with higher threshold than in MACRO and Baksan 
shows less profound 6 - effect, although some suppression of the fiux is observed both at 
cos© = —0.6 ^ — 0.2 and in the vertical bins. 

In [0 it was observed that oscillations of muon neutrinos to sterile neutrinos, ^ Ug, 
lead to the zenith angle dependence being qualitatively similar to that observed by MACRO 
experiment. In particular, it has been shown that the narrow dip in vertical bins can be 
due to the parametric enhancement of oscillations for neutrinos whose trajectories cross 
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both the mantle and the core of the Earth. The oscillation effect decreases with increase 
of the neutrino energy. This can explain absence of strong deficit in experiments with high 
thresholds like the Super-Kamiokande. 

The z/^ Us oscillations give as good description of the low energy (sub-GeV and multi- 
GeV) data as i/^ ^ do (see e.g. |^). The appropriate region of oscillation parameters 

Am^ = (2 ^ 10) ■ 10~^ eV^ sin^ 20 > 0.8 (1) 

is shifted slightly to larger values of Am^ in comparison with the — > v^- case due to matter 
effect which becomes important for z/^ Vg channel at small Am^. 

Crucial difference between Vs and Vj. channels appears for high energy 

atmospheric neutrinos which are detected by upward-going muons. The matter effect is 
absent in z/^ ^ v-r channel but it is important for ^ Us 

In this paper we perform a detailed study of the zenith angle dependences of the at- 
mospheric neutrino and muon fluxes in presence of the z/^ ^ z/^ oscillations. We give a 
description of the parametric resonance effect in sect. 2. In sect. 3 we present results of 
calculations of the zenith angle dependences for the through-going and stopping muons. In 
sect. 4 we compare the prediction with experimental results. 

2. Parametric resonance in atmospheric neutrinos 

The z/^ ^ Us oscillations in matter are described by the 2x2 evolution matrix (the 
effective Hamiltonian) with off-diagonal elements if^s = Hsfj. = Am^/AE ■ sin 26' and the 
difference of the diagonal elements: 

Attz^ 

H^^-Hss = ^cos20 - V . (2) 

Here 9 is the vacuum mixing angle, Am^ = m\ — m\ is the mass squared difference (m2 is the 
mass of the neutrino component which has larger admixture in in the case of non-maximal 
mixing), E is the energy of neutrino and V is the matter potential 

y = ^Nn , (3) 
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where Gp is the Fermi constant, Nn is the neutron number density Q (For antineutrinos V 
has an opposite sign). 

According to (0, ^ the length of z/^ ^ Ug oscillations in matter and the effective mixing 
angle 6m are determined by 
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(^cos2e + l)2 + e^sin2 2^ , (4) 



sin' 2^^ = sin' 20^' (^cos20 + 1)' + ^'sin'2^ , (5) 
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where 



2EV 

In the case of maximal mixing the equations (0) and (B) are simplified: 
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For Am' > 4 ■ 10"^ eV' both for the sub-GeV and muhi-GeV ~ 3 ^ 5 GeV) events 
we get ^' > 10 and the matter effect is small (< 10% in probability). Thus, in significant 
region of parameters (masses and mixing) at low energies the z/^ Us oscillations reproduce 
results of the Uj. oscillations, as far as the charged current interactions are concerned. 

(The number of the tau-lepton events is small.) 

The matter effect becomes important for multi-GeV events if Am' < 3 • 10~^ eV' 
In particular, the zenith angle dependence of ratios is modified. Since matter suppresses the 
i^fi ^ oscillations, one expects weaker oscillation effect for the upward-going neutrinos, 
and therefore, flatter zenith angle dependence than in the ^ case where matter effect 
is absent. 



* Notice that in general case of mixed with v-r and z^e, the off-diagonal (mixing) elements get 
contributions from interaction with matter, and also the potential depends on electron/proton 
density. 
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The matter effect is important for distributions of the through-going and stopping muons 
produced by high energy neutrinos even for large Am^. 

Let us first describe the zenith angle dependence of the survival probability Pl^fj, — »• 
z/^) for fixed neutrino energy. To a good approximation the Earth can be considered as 
consisting of the core and the mantle with constant densities and sharp change of density 
at the border between the mantle and the core. Therefore propagation of neutrinos through 
the Earth has a character of oscillations in layers with constant densities. This gives not 
only correct qualitative picture but also rather precise (as we will see) quantitative results. 

According to (|) with increasing energy (decreasing Am^) ^ — > and the oscillation 
length increases approaching the asymptotic value determined by the potential only: 1^ ~ 
2tt/V. At the same time, the effective mixing angle decreases (|]), so that the oscillation 
effects become weaker. Let us consider the range of energies which corresponds to ^ = 0.1-i-l. 
Here Im weakly depends on ^, but sin^ 29m is not yet strongly suppressed. In this case the 
phase of oscillations, $, acquired by neutrino in a given layer AL equals: 

^ = 2ti dLV . (7) 



The phase $ depends on density and size of the layer, and it does not depend on neutrino 
energy. Using this feature we can immediately get the zenith angle dependence of Piv^i 

v^). Notice that for cosO = —0.8 neutrino trajectories touch the core of the Earth. 
Therefore for cosB > —0.8 neutrinos cross the mantle only, whereas for cosO ^ —0.8 they 
cross both the mantle and the core. 

For COS0 > —0.8 (mantle effect only), the survival probability can be written as 

P^l-sin2 2e„sin2i^, (8) 
where the phase acquired by neutrinos, $m(9), equals: 

$™(0) ^ 2i?i5|cose|V ^ V2GFNnRE\cosQ\ . (9) 
Here Re is the radius of the Earth. From (^) we find $(cos G = —0.4) = vr, so that maximal 



oscillation effect, Pmax = 1 — sm'^29m, is at cos 6 = —0.4. At cos 6 = —0.8 the phase is 
$m = 27r, and the oscillation effect is zero. 

For cosO < —0.8, neutrinos cross three layers: mantle, core and again mantle. The 
survival probability is P = jS'^^p, where the evolution matrix 5* equals 

s = uidjDi<^^)u+id^ - e,)D{^,)u{e^ - ^jD($^)f/+(^„) . (lo) 

Here U{6m) is the 2x2 mixing matrix, 6m and 6c are the mixing angles in matter of the 
mantle and core correspondingly. The matrix 

D($) = diag{l,e''^) (11) 

describes the evolution of the eigenstates in certain layer; $c and are the phases of 
oscillations acquired in the core and in each layer of the mantle. It turns out that ~ ^• 
Moreover, for cos© ~ —0.88 also <l>c equals vr. Thus for cos© ~ —0.88 we get the equality: 

$ml «i $c ~ $m2 ~ TT . (12) 

Under this condition an enhancement of oscillations occurs. Indeed, now D = diag{l, —1) 
for all layers, and inserting this D in (pUj), we get the probability 

P= |S'^^|' = l-sin2(4^^^-2^e). (13) 

Clearly for 6c < 6m, the probability P is smaller than 1 — sin^ 26m and 1 — sin^ 26c corre- 
sponding to maximal oscillation effect in one density layer. Moreover, the smaller 6c (the 
stronger suppression of mixing in the core) the bigger the transition effect. 

The condition ([T^) means that the size of the layer, R, coincides with half of the oscil- 



lation length: R = lm/2. This is the condition of the parametric resonance |TO| , p]l| which 
can be written in general as, 27r Jq^ dr/lm = 27ik, k = 1,2,3..., where rj is the size (pe- 
riod) of perturbation, and in our case Vf = Rc + Rm- Graphical representation [jlT| of the 
enhancement is shown in fig. 1: Mixing angle changes suddenly when the phase of system 
reaches vr. This leads to increase of the oscillation angle. Thus one expects the (parametric 



resonance) peak at cos0 = —0.88. The width of the peak is AcosG ~ 0.12. For exact 
vertical direction cos0 = — 1 , we get $c ~ 2.57r and the enhancement is destroyed. Thus 
unsuppressed flux is expected in this direction. 

Let us stress that for sufficiently large energies the equality ([T2| ) does not depend on 
neutrino masses. The equality is determined basically by the density distribution in the 
Earth and by the potential which, in turn, is fixed by the channel of oscillations and the 
Standard Model interactions. The equality is fulfilled for oscillations into sterile neutrinos 
only. In the i/^ ^ v^. channel the potential is two times larger and equality ([T^) is broken. 

With the increase of the neutrino energy the form of the zenith angle dependence (po- 
sition of minima and maxima) practically does not change, however the mixing angle, and 
therefore the depth of oscillations, diminish. Oscillation effect disappears for ^ <^ 0.1. In 
contrast, for low energies, ^ > 1, it reduces to the vacuum oscillations effect. Thus the pro- 
found zenith angle dependence with two dips exists in rather narrow range: ^ ~ 0.1 -r- 0.5 
(fig. 2). This dependence coincides qualitatively with the zenith angle dependence of the 
muon fiux observed by the MACRO experiment. The G-distributions in fig. 2 have been 
calculated for real density profile of the Earth . They close to the results obtained from 
(|) ( pUP for simplified model of the Earth (layers with constant density) (see fig. 2 in [|^). 



3. Zenith angle distribution of the upward-going muons 

The fiux of muons with energy above the threshold energy E^^^ as a function of the zenith 
angle equals 

F^iE'l^, 9) = \ dE-Y. Fr^E, 9) ■ P,(i?, 9) ■ Y,{E, E^'), (14) 

M l = U,U 

where Fy{E, 9) is the fiux of original neutrinos, P{E, 9) is the survival probability, and 

y.{E, Ef) = /^^^ R{E'^, Ef) . ^{E, E'^) (15) 

is the probability that the neutrino with energy E produces muon, which arrives at a detector 
with the energy above E^^. Here R{E'^,Ejl^) is the muon range (the distance in g/crn? 
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at which muon energy decreases from E'^ to Ejj^ due to energy loss) and da/dE'^ is the 
differential cross section of v^N reaction. 



The integrand in (|1^ at P = 1: 

/ = F{E,Q)-Y{E,E';^) (16) 

describes the energy distribution of neutrinos that give rise to muon flux above E^^ . It has 
the form of a wide peak with full width on the half of height characterized by 1.5^2 orders 
of magnitude in E. This is substantially bigger than the width of region with significant 
two dips effect = 0.1 ^ 0.5). A position of maximum of the peak, E^"'^, depends on Ejj^ 
and e, and for Ejj" ~ 1 GeV one has E™"^^ ^ (20 ^ 30) GeV. 

Due to integration over neutrino energies the - distribution of muons differs from the 
distribution of neutrinos shown in fig. 2. There are two modifications: 

(i) Smoothing of O dependence occurs due to contributions from low energy neutrinos 
(^ > 1) which undergo basically averaged vacuum oscillations effect. This leads to decrease 
of the peak between two dips in 9-dependence. 

(ii) Weakening of the overall suppression of the flux occurs due to contribution from high 
energy neutrinos with ^ < 0.1. For these neutrinos the oscillation effect is small due to 
smallness of mixing angle. 

The dependence of the ratio of fluxes with and without oscillations F^/ F^ on O for differ- 
ent values of Am^ is shown in fig. 3a. With increase of Am^ the region of strong oscillation 
effect {P{E)) shifts to higher energies and thus larger part of the neutrino spectrum will 
undergo suppression. This leads to overall strengthening of the oscillation effect, however 
the contribution from low energies also increases and the two dips zenith angle dependence 
becomes less profound. In contrast, for small Am^ the 6-dependence is more profound, but 
overall suppression is weaker. Similar changes occur when the threshold, Ejj^, varies (fig. 
3b). With increase of Ejj^ (for fixed Am^) the peak of the integrand (p!6D shifts to higher 
energies; the G-dependence with two dips becomes more profound but overall suppression 



effect weakens. This can explain weaker effect in experiments with higher Ejj^ (like the 
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Super-Kamiokande) . (Note that there is no E^^ / scahng: the change of Am^ leads to 
stronger effect than similar change of 

For non-maximal mixing a dependence of the oscillation length on the neutrino energy 
becomes stronger: it appears in the lowest order in C,- Indeed, from (^) we have 

2n 1 
™ y lTe-cos2^ ' 

where for Am^ > minus and plus signs are for neutrinos and antineutrinos correspondingly. 
For non-maximal mixing the oscillation length of neutrinos is larger and the condition of 
the parametric resonance is fulfilled for larger | cosO|. Thus, the parametric peak shifts to 
vertical direction. For antineutrinos the length is smaller and the effect is opposite. Since 
the i/-flux is larger than z/-flux the parametric peak for total flux shifts to larger | cos G| (fig. 
3c). 

Uncertainties in the shape of the neutrino energy spectrum (and cross sections) can 
influence the zenith angle dependence. To study this effect we parameterize possible modi- 
fications of the spectrum, and consequently the integrand I as 

I{E) = /o(E)AE"+'^i"(^/^^^v\ (17) 



where Io{E) is the integrand which corresponds to the spectrum from ||T3|, a and (3 are 
parameters and A is a normalization factor. The results of the calculations for different values 
of parameters a and f3 are shown in fig. 3d. As follows from fig. 3, shift of the maximum to 
higher energies leads to suppression of the vacuum oscillation effect and therefore to more 
profound zenith angle dependence (dashed line), however the overall suppression becomes 
weaker. In contrast, the shift of the maximum to lower energies enhances the vacuum 
oscillation effect (dashed dotted line). Both more profound zenith angle dependence and 
strong overall suppression can be obtained for narrower I{E) distribution (dotted line). 

For stopping muon sample, the peak Istop{E) is narrow and its maximum is at smaller 
energies: E ~ 10 GeV. For Am^ > 5 ■ 10^^ eV^ the matter effect is small (< 10%), so 
that the zenith angle distribution is determined basically by vacuum oscillations (fig. 4). 
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Difference between z/^ — > z/^ and z/^ Vr channels is small. For Am^ < 4 ■ 10~^ eV^ the 
matter effect becomes important. The zenith angle dependence acquires the form with two 
dips. 

Similarly, the sample of muons produced in the detector itself corresponds to low energies 
and narrow peak of the intergand, thus leading essentially to vacuum oscillations effect. So, 
we expect strong suppression of the number of these events which weakly depends on zenith 
angle. 

4. Predictions versus experimental results 

Let us compare the predicted distributions with the experimental results from the Bak- 
san, MACRO (fig. 5a) and Super-Kamiokande experiments (fig. 5 b). Histograms show the 
0-distribution averaged over the bins for "probe" point Am^ = 8 ■ 10~'^ eV^, sin^ 26 = 1, and 



neutrino spectrum from ||T3[. The predicted suppression in the vertical bin is still weaker 
than in MACRO but stronger than in Baksan data. The predicted curve would fit better 
average of MACRO and Baksan results. Integrations over neutrino and muon energies result 
in strong smoothing of the distribution. The difference of suppressions in the dips and in 
the region between the dips is about (15 20) %. Without parametric effect one would 
expect the ratio F^/F^ — 1 in the vertical bins. Notice that still error bars are large and the 
experimental situation for vertical bins is controversial (fig. 5a). In the same time there is 
a good agreement of predictions with the Super-Kamiokande zenith angle dependence (fig. 
5b). Clearly, more data are needed to make any conclusion. 

As follows from fig. 3, varying the neutrino parameters (Am^, sin^ 2^) and modifying 
neutrino spectrum one can change the predicted G-dependence. In particular, it is possible 
to make the dips move profound, to change relative depth of the dips, to shift positions of 
minima in cosG. Fitting the data one can also take into account uncertainties in normal- 
ization of original neutrino flux which can reach 20%. 

Clearly G- dependences for —>■ and Vj. channels are different. The z/^ — *• 

Vj. oscillations lead to smooth enhancement of suppression with | cos G | , whereas the z/^ 
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Us oscillations result in the structure with two dips. Furthermore, for vertical bins a 
suppression due to Uj. is stronger than due to z/^ — » and the difference can be as big 

as 20%. For z/^ Ur channel suppression weakly depends on the energy threshold, so that 
for Baksan/MACRO and Super-Kamiokande experiments the dependences are similar. In 
contrast, for Vg channel the suppression effect decreases with increase of the threshold. 
The effect is large for stopping muons (or muons produced in the detector), it is weaker for 
through-going muons in Baksan/MACRO and the weakest effect is for through-going muons 
in the Super-Kamiokande. 

The two channels can be distinguished also by studying neutral current effects 0]. 

In conclusion, in the range of Vg oscillations parameters, which correspond to the 

best fit of the low energy data, the zenith angle dependence of the upward-going muons has 
a peculiar form with two dips at cosO = (—0.6^ — 0.2) and cosO = (—1.0^ — 0.8). The dip 
in vertical direction is due to parametric resonance in oscillations of neutrinos which cross 
the core of the Earth. This zenith angle dependence is crucial signature of a solution of the 
atmospheric neutrino problem, based on Vg oscillations. 
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Figure Captions 



Fig. 1. Graphical representation of the parametric enhancement of neutrino oscillations. 
Vector IJ' describes the neutrino state in such a way that its projection on the axis Y gives 
2P — 1, where P is the probability to find j/^-neutrino. (The axis X is 'Re{%p*^%l)s)-, where ip^ 
and ips are the wave functions of the muon and sterile neutrinos.) The positions of i/- vector 
2-4 correspond to //-states at the borders of mantle and the core. 

Fig. 2. The zenith angle dependence of the survival probability P( ^ v^) for different 
neutrino energies: solid line - 100 GeV, dashed line - 50 GeV, dash-dotted line - 30 GeV 
(Am2 = 8 ■ 10-3 eV2). 

Fig. 3. The zenith angle dependence of the ratio F^/ for the upward-going muons. a), 
the G dependences for different values of /S.rn? (sin^ 29 = 1 and Ejj^ = 1 GeV); b). the same 
for different energy thresholds of muon detection, Ejj^ (sin^ 26' = 1 and Am^ = 8 ■ 10"^ eV^); 
c). the same for different values of mixing angle (Am^ = 8 ■ 10-^ eV^ and Ejj^ = 1 GeV); d). 
the same for different original spectra of neutrinos described by parameters a, (3 in (jl^: 
a = 0.75, (3 = -0.075 (dashed line) a = 0.25, (3 = -0.075 (dash-dotted line) a = 0.75, f3 = 
-0.10 (dotted line), solid line corresponds to unmodified neutrino flux. 
Fig. 4. The zenith angle dependence of the ratio F^/F^ for stopping muons for different 
values of Am^ (sin^ 29 = 1 and Ef = 1 GeV). 

Fig. 5. Comparison of the predicted zenith angle dependence (histograms) with experi- 
mental data from the MACRO and Baksan (a) and Super-Kamiokande {Ejl^ = 7 GeV) (b) 
experiments. Solid lines for oscillations with Am^ = 8 ■ 10"^ eV^ and sin^ 29 = 1; 

dashed line for z/^ — > Ur oscillations with Am^ = 5 ■ 10-^ eV^ and sin^ 29 = 1. 
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